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1.  Introduction  and  summary 

The  main  result  of  this  paper,  theorem  4,  is  that  there  exists  an  infinite 
sequence  of  binary  digits  which  can  be  used  to  generate  sliding  parity  check 
codes  for  the  binary  symmetric  channel  (BSC),  and  the  error  probability  for 
such  codes  is  close,  in  a  certain  sense,  to  minimum  uniformly  in  all  of  the  param¬ 
eters:  block  length,  probability  of  error  for  a  single  digit,  and  rate  not  too  much 
less  than  capacity.  This  result  is  obtained  by  studying  the  metric  structure  which 
can  be  required  of  codes,  and  then  relating  this  to  the  error  probability.  The 
paper  is  essentially  self-contained. 

Let  Bn  be  the  set  of  2n  ordered  n-tuples  from  B  =  Bl  =  {0,  1}  where  an  ele¬ 
ment  of  B  is  called  a  binary  digit  or  bit.  For  any  1  we  define  a  metric  on 
Bn  by  ab  =  the  number  of  coordinates  in  which  a  and  b  differ.  An  n-code  Cn  is  a 
nonempty  subset  of  Bn ;  an  element  c  £  Cn  is  called  a  codeword.  A  code  is  any 
set  which  is  an  n-code  for  some  n.  The  code  Cn  in  a  sequence  of  codes  Ci,  C2, 
is  always  an  n-code.  The  probability  law  of  a  BSC  is  defined  as  follows:  for  each 
a  G  Bn  there  is  a  probability  distribution  on  Bn  given  by 

(1)  P{b\o]  =  pabqn~ab , 

where  0  <  p  <  1/2  and  q  =  1  —  p.  A  statement  which  is  made  for  all  p  means 
for  all  p  such  that  0  <  p  <  1/2.  Here  P{b\a}  is  the  probability  that  the  channel 
output  is  b  given  that  the  channel  input  is  a.  When  w  =  lwe  have  .P{0|0}  = 
P{1|1}  =  q)  P{1|0}  =  P{0|1}  —  p  so  that  the  probability  p  of  an  error  in  a 
single  digit  does  not  depend  on  what  that  digit  is,  hence  the  “symmetric” 
designation. 

A  decoder  for  an  n-code  Cn  is  a  set  1)  of  disjoint  subsets  of  Bn  and  a  1  to  1 
correspondence  between  C„  and  D.  Assume  for  the  moment  that  the  elements 
of  Cn  and  D  are  ordered  by  indices  so  that  Cn  =  (ci,  *  •  •  ,  c8} ,  D  =  {Dh  •  •  •  ,  Ds } 
and  ck  corresponds  to  Dk  under  the  1  to  1  correspondence.  In  application,  the 
sender  and  receiver  first  decide  on  Cn  and  D  and  then  use  these  repeatedly.  The 
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sender  feeds  a  c*  £  (7n  into  the  channel  and  the  receiver  observes  b  £  Bn  at  the 
output  with  probability 

(2)  pCk5qn-CkS' 

If  b  £  Dj  then  the  receiver  decides  that  c3  was  sent.  Thus  the  probability  that 
the  receiver  makes  an  error,  if  c*  is  sent,  is  P{Dk\ck }.  Let 

(3)  ep(Cn)  =  min  max  P{D£\ck}. 

D  k 

Thus  ep(Cn)  is  the  smallest  probability  of  error  that  can  be  guaranteed  for  all 
codewords  of  Cn  if  a  best  decoder,  for  this  purpose,  is  used  and  the  probability 
of  a  single  digit  being  received  in  error  is  p. 

The  rate  Rn  =  R(Cn)  of  an  n-code  Cn  is  defined  by  Rn  =  (1  /n)  log  (the  number 
of  codewords  in  Cw),  where  all  logarithms  are  to  the  base  2,  so  that  Cn  has  2nR* 
codewords.  The  function  H{x)  =  —x  log  x  —  (1  —  x)  log  (1  —  x),  H( 0)  =  0, 
H{  1/2)  =  1,  is  restricted  to  the  interval  0  ^  x  g  1/2  so  that  it  is  continuous 
and  strictly  increasing  and  has  a  unique  inverse.  The  well-known  channel 
capacity  for  the  BSC  is  1  —  H{p). 

One  of  the  results  obtained  is  an  upper  bound  for  ep(Cn).  The  decoder  used 
for  this  purpose  associates  to  c*,  the  set 

(4)  Dk  =  {b\ckb  ^  npn  and  c*!>  <  Fjb  for  all  j  5*  k}, 

where  p'n  is  defined  by  Rn  =  1  —  H(p'n).  Thus  Dk  consists  of  those  b,  within  a 
sphere  of  radius  np^  of  ck,  which  are  strictly  closer  to  ck  than  to  any  other  code¬ 
word.  Note  that  this  decoder  depends  only  on  Cn  and  not  on  p.  Clearly  these  Dk 
are  disjoint,  so  that 

(5)  ep(Cn )  ^  max  P{Di\ck}. 

k 


In  this  notation  Shannon’s  coding  theorem  for  the  BSC  is 
Theorem  1.  (a)  Direct  half :  For  any  p  and  R  satisfying  0  <  R  <  1  —  H(p) 

there  exists  a  sequence  of  codes  Ci,  C2,  *  •  *  such  that  R(Cn)  — >  R  and  ep(Cn )  — >  0. 

(b)  Converse  half :  For  any  p  and  R  >  1  —  H(p)>  if  Ci,  C2,  *  •  •  is  a  sequence 
of  codes  with  R(Cn)  — >  R ,  then  ep(Cn )  1. 

Our  concern  is  only  with  the  direct  half  of  the  theorem.  Now  ep(Cn )  can  be 
made  to  go  to  zero  exponentially  and  the  best  exponential  error  rate  is  known 
for  a  certain  interval  below  capacity.  Theorem  2  gives  a  lower  bound  for  the 
limiting  behavior  while  theorem  3  states  that  this  lower  bound  is  attainable,  in 
the  limit,  for  a  certain  interval  of  rates  below  capacity.  This  interval  is  given  by 


(6) 

where 

(7) 


<R  =  1  -  H{pf)  <  1  -  Hip), 


so  that  the  interval  is  always  nonempty.  The  parameter  p'}  defined  in  terms 
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of  R,  is  a  useful  one  for  the  statement  of  the  result.  For  later  convenience  the 
sequence  of  codes  in  theorem  3  starts  with  C2. 

Theorem  2.  For  any  p  and  R  satisfying  0  <  R  =  1  —  H(p')  <  1  —  H(p), 
if  Ci,  C2,  •  •  •  is  a  sequence  of  codes  satisfying  R(Cn)  — >  R,  then 

(8)  lim  inf  Q)  log  eP(Cn)  ^  H(p’)  +  p'  log  p  +  (1  -  p')  log  q  <  0. 

Theorem  3.  For  any  p  and  R  satisfying  (6)  there  exists  a  sequence  of  codes 
C2j  C3,  •  •  •  such  that  R(Cn)  — >  22  and 

(9)  lim  (^  log  ep{Cn)  =  H(p')  +  p'  log  p  +  (1  -  p')  log  q  <  0. 

If  f{p)  =  H(p')  +  p'  log  p  +  (1  —  p')  log  q ,  then  f(p')  =  0  and  [the  natural 
logarithm  of  2][(df)/(dp)]  =  ( p '  —  p)/(p?)  >  Oifp  <  p'.Now  H(p)  <  H(p')  in 
theorems  2  and  3  and  this  implies  that  p  <  p'  so  that  f(p)  <  0  as  stated. 

Our  interest  is  in  theorem  3  and  in  certain  ways  that  the  sequence  of  codes 
C2,  C3,  •  •  •  can  be  restricted  and  still  make  theorem  3  true.  A  restricted  class  of 
codes  of  interest  is  the  parity  check  (PC)  codes.  Given  an  integer  n  and  a  num¬ 
ber  Rn,  0  <  Rn  <  1,  such  that  nRn  =  k  is  also  an  integer,  an  n-PC  code  Cn, 
with  rate  Rn ,  is  an  n-code  which  can  be  generated  from  some  matrix  a,j, 
i  —  1,  •  •  •  ,  n  —  k;j  =  1,  •  •  •  ,  k\  where  each  a#  E  B.  The  2k  codewords  of  Cn 
are  generated  as  follows:  given  any  (bh  •••,&*)£  Bk ,  extend  it  to  Bn  by 
determining  b*+i,  •  •  •  ,  bn  from 

k 

(10)  bk+i  ~  O'ijbj  i  —  •••,?!“■  k 

3  =  1 

where  mod  2  arithmetic  is  used,  that  is,  0  0  =  0*1  =  10  =  0,  1*1  =  1,0  + 
0  =  1  +  1=  0,  0  +  l  =  l  +  0  =  l.  The  codewords  of  C»  are  the  2k  n-tuples 
(6i,  •  •  •  ,  bn)  obtained  in  this  way  as  (bh  •  •  •  ,  bk)  ranges  over  Bk.  A  special  class 
of  PC  codes  is  sliding  parity  check  (SPC)  codes.  If  the  generating  matrix  of  an 
n-PC  code  satisfies  a,*>  =  a»+j_i,  where  ah  •  •  •  ,  a^i  is  a  given  sequence,  the 
resulting  code  is  an  n-SPC  code.  Thus,  any  binary  sequence  «!,•••,  an-i  deter¬ 
mines  an  n-SPC  code  for  every  rate  Rn  =  1/n,  •  •  •  ,  (n  —  l)/n. 

Theorems  2  and  3,  and  the  fact  that  theorem  3  remains  true  if  the  codes  are 
required  to  be  SPC  codes,  were  proved  by  Elias  [1],  [2],  who  also  obtained 
bounds  on  the  probability  of  error.  Proofs  of  theorems  1,  2,  3  and  a  modified 
form  of  Elias’  proof  showing  that  theorem  3  remains  true  if  the  codes  are  required 
to  be  PC  codes  can  be  found  in  Feinstein  [3]. 

The  advantages  of  using  PC  codes  in  the  physical  implementation  of  the 
coding  operation  are  fairly  obvious.  An  n-SPC  code  requires  only  the  storage 
of  (n  —  1)  bits,  an  n-PC  code  requires  only  the  storage  of  (1  —  Rn)nRnn  g  n2/4 
bits,  and  a  general  n-code  requires  the  storage  of  n2nRn  bits.  If  Rn  is,  say,  1/2, 
and  n  is  moderately  large,  say  100  or  1000,  then  a  general  n-code  would  require 
an  astronomical  amount  of  storage,  while  an  n-SPC  code  requires  only  a  reason¬ 
able  amount  of  storage,  and  some  simple  algebraic  computations  when  in  use. 
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Thus  the  implementation  of  the  coding  operation  for  an  n-SPC  code,  once 
ah  •  •  • ,  a»_i  have  been  obtained,  seems  to  be  feasible  for  moderately  large  n;  how¬ 
ever,  the  exhaustive  method  of  examining  all  2n_1  such  sequences  of  ah  •  •  •  ,  an~  1 
and  then  selecting  the  best  one,  is  clearly  not  a  workable  method  for  finding  a 
satisfactory  ah  •  •  •  ,  an_i.  Very  little  is  known  in  this  direction. 

Now  to  say  that  theorem  3  remains  true  when  the  codes  are  required  to  be  SPC 
codes  means  that  for  every  p  and  R  satisfying  (6)  there  exists  a  sequence 
d2,  c?3,  •  •  •  ,  where  dn  £  Bn~xy  such  that  the  corresponding  sequence  of  SPC 
codes  satisfies  (9)  and  Rn  — >  R.  A  natural  question  to  ask  is  whether  there  exists 
one  infinite  sequence  a  =  (ah  a2,  •  •  •)  such  that  we  can  let  dn  =  (ah  •  •  •  ,  an_i). 
It  would  be  better  yet  if  this  same  a  could  be  used  for  all  p  and  R  satisfying  (6). 
This  will  be  shown  to  be  true  (theorem  5).  From  a  practical  point  of  view  this 
fact  still  leaves  the  concern  that  any  such  a  might  produce  only  terrible  codes 
for  moderate  n.  Thus  what  we  do  (theorem  4)  is  prove  that  there  exists  an  a 
such  that  any  SPC  code  Cn  constructed  from  it  has  its  ep(Cn)  bounded  above 
by  a  certain  simple  function  of  n,  Rn)  and  p\  and  this  bound  has  the  obvious 
desired  asymptotic  behavior  for  any  Ry  p  satisfying  (6).  To  the  extent  that  the 
error  bound  is  satisfactory,  and  assuming  that  the  restriction  on  Rn  is  not  objec¬ 
tionable,  the  result  says  that  there  is  one  a  which  can  be  used  to  design  satis¬ 
factory  SPC  codes  for  all  parameter  (n,  Rn>  p)  values  for  the  BSC.  Unfortunately, 
we  cannot  exhibit  such  an  a. 

The  method  of  proof  has  several  advantages  in  addition  to  being  fairly  simple 
and  making  the  results  more  transparent.  For  one  thing,  if  a  particular  n-PC 
code  Cn  is  being  considered,  one  can  compute  a  certain  function,  Nd,  and  then 
immediately  obtain  a  bound  on  ep(Cn ),  and  also  immediately  make  statements 
like  the  following:  “Regardless  of  which  codeword  c  is  the  input,  the  receiver 
will  not  make  a  mistake  if  cb  ^  34,  where  b  is  the  received  n-tuple;  also,  the 
receiver  will  make  a  mistake  in  at  most  2  per  cent  of  the  cases  where  cb  =  41.” 
The  more  usual,  and  much  longer,  computational  approach  is  first  to  get  all  of 
the  data  similar  to  that  in  the  previous  sentence,  and  then  compute  ep(Cn). 
Statements  like  the  quoted  one  are  particularly  powerful  in  that  they  don’t 
depend  on  the  probability  law  of  the  binary  channel  and  so  have  some  use  even 
when  the  errors  are  dependent  and  nonstationary. 

Let  5  be  the  class  of  infinite  sequences  a  =  (ah  a2,  *  •  •)  where  each  a*  £  B. 
Let  Q  be  the  probability  measure  on  the  usual  a-field  of  subsets  of  25  which 
makes  the  coordinate  random  variables  independent  and  uniformly  distributed, 
that  is,  Q{a|ai  =  aj,  a2  =  a2,  •  •  •  ,  a„  =  a'n}  =  (l/2)n.  The  principal  result  is 

Theorem  4.  For  any  13  >  1  there  is  a  set  Bp  (ZB  with  Q(Bp)  ^  1  —  (1/0) 
such  that  for  every  a  —  (ah  a2,  •  •  •)  £  Bp  the  following  is  true :  For  any  p ,  Rny  n  ^  2 
such  that  nRn  is  an  integer  and 

(11)  1  -  H  (  .-Vp  <  Rn  =  1  -  H(p'n)  <  1  -  H(p), 

\  v  p  +  V  q/ 

the  ri-SPC  code  Cn  with  rate  R„  which  is  generated  from  ai,  •  ■  ■  ,  an- 1  satisfies 
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(12)  i  log  ev{Cn)  S  l  [log  (*  §)  +  5  log  »] 

+  H(pfn)  +  pn  log  p  +  (1  -  p'n )  log  q. 

Q{B&}  >  0  so  that  B$  is  not  empty;  and  since  the  first  term  on  the  right 
side  of  (12)  goes  to  0  as  n  goes  to  infinity,  such  a  sequence  of  codes  satisfies 
theorem  3. 

By  taking  (5  =  2,  3,  •  •  •  and  letting  F  =  UJ°=2  Bi}  it  is  clear  that  theorem  4 
immediately  implies 

Theorem  5.  There  is  a  set  F  CB  with  Q(F)  =  1  such  that  for  '  any  a  = 
(ah  a2,  •  •  •)  and  any  p,  72  satisfying  (6),  the  following  is  true :  //  {/2n}  is  a 
sequence  of  rates  with  Rn  — » R,  and  {nRn}  is  always  integer -valued,  and  Cn  is  the 
n-SPC  code  of  rate  Rn  generated  from  (ax,  •  •  •  ,  an_i),  ^cn  (9)  holds. 

Thus  if  an  a  £  B  is  selected  at  random,  according  to  the  Q  distribution,  then 
with  Q-probability  1  it  will  generate  an  asymptotically  optimum  sequence  of 
SPC  codes  for  every  p,  R  satisfying  (6).  However,  Q(F )  =  1  implies  that  there 
is  at  least  one  a  £  F  with  only,  say,  one  1  in  its  first  1000  coordinates,  and  it  is 
easy  to  show  that  the  n-SPC  codes  such  an  a  generates  for  n  ^  1000  are  usually 
terrible.  Such  possibilities  make  a  result  like  theorem  4  more  useful  than  theo¬ 
rem  5. 


2.  The  existence  of  codes  with  certain  metric  properties 


It  is  intuitively  reasonable  that  a  code  whose  codewords  are  far  apart  should 
be  a  good  code.  Given  an  n-code  Cn  =  {ci,  •  •  •  ,  c8}  and  an  integer  d,  1  ^  d  ^  n, 
let  Nd(cm)  equal  the  number  of  codewords  whose  distance  from  codeword  cm  is  d. 
The  existence  of  codes  with  specified  upper  bounds  for  Nd(cm )  will  be  proved, 
and  then,  in  lemma  3  and  the  proof  of  theorem  4,  such  bounds  will  be  converted 
into  upper  bounds  on  ep(Cn ). 

There  is  a  simple  way  to  guess  at  the  nature  of  the  bounds  that  can  be 
obtained  for  Nd(cm ).  Say  that  we  wish  to  construct  an  n-code  with  rate  Rn, 
and  attempt  to  do  so  by  selecting  2nR*  elements  of  Bn  at  random.  If  cm  is  the 

mth  codeword  selected  then  there  is  probability  ^^(l/2n)  that  any  particular 

one  of  the  2nRn  —  1  other  selections  will  result  in  a  codeword  at  distance  d 
from  Cm.  Thus  the  expected  number  of  codewords  at  distance  d  from  cm  is  about 

2n/e"(d)(l/2n),  anc*  this  is  the  form  of  the  bound  that  we  will  obtain  for  Nd{cm). 

Define  ph  by  Rn  =  1  —  H(pn)',  then  an  elementary  fact,  lemma  4b,  implies  that 


(13) 


2  [H(d/n)-H(p'n)n] 


and  by  use  of  Stirling’s  formula  this  can  be  converted  into  an  approximate 
equality  for  large  n .  This  indicates  that  the  parameter  np'n  may  be  interpreted 


674 


FOURTH  BERKELEY  SYMPOSIUM:  THOMASIAN 


as  the  distance  d  at  which  Nd(cm)  becomes  significant,  if  the  code  is  selected  at 
random. 

One  of  the  nice  properties  of  PC  codes  is  that  if  Cn  =  {ci,  •  •  •  ,  c,}  is  an  n-PC 
code  then  Nd(cm)  is  independent  of  m.  In  order  to  prove  this,  define  the  weight 
w(b)  of  an  element  b  £  Bn  to  be  the  number  of  coordinates  of  b  which  equal  1. 
If  b  =  (&i,  •  *  •  ,  bn)  and  b'  =  (bi,  •  •  •  ,  b'n)  are  elements  of  Bn  then  let  b  +  b'  = 
(bi  +  bi,  •  •  •  ,  bn  +  K)  where  the  addition  is  mod  2.  Clearly  there  will  be  a  con¬ 
tribution  of  1  to  w(b  +  b')  from  the  zth  coordinate  of  (b  +  b')  if  and  only  if 
bi  bt,  so  that  W  =  w(b  +  6').  Thus  Nd(cm)  equals  the  number  of  times  that 
w(cm  +  c)  equals  d  when  c  ranges  over  Cn  precisely  once.  Now  let  Cn  be  an  n-PC 
code  of  rate  k/n  generated  by  a  matrix  a Clearly  0  =  (0,  •  •  •  ,  0)  £  Cn  and 
Nd(0)  —  Nd  =  the  number  of  codewords  of  weight  d  —  the  number  of  times  that 


,  bk ,  2^  ciijbj, 

i-i 


,  X)  Cln—k,jbj\ 

y-i  / 


equals  d ,  as  (bi,  •  •  •  ,  bk)  ranges  over  Bk  precisely  once.  Let  cm  be  a  particular 
codeword  generated  by,  say,  (bi,  •  •  •  ,  b*),  then  Nd(cm)  is  the  number  of  times 
that 


| bi  +  bi, 


k  k 

,  bk  +  bi,  £  a\jbj  +  £  O'ljbjy 

y-i  y- 1 


*  *  \ 

^  Q'n—kfjbj  “|“  ^  (In—ktjbj  I 

y-i  y-i  / 


=  w  ^ 


bi  +  bi, 


,  bfc  +  ]£  au(b;  +  bj), 

y-i 


,  S  0'n~k,j{bj  fy) 

1  > 


equals  d  as  (bi,  •  •  •  ,  bfc)  ranges  over  B*  precisely  once.  However  (bi  +  bj,  •  •  •  , 
bk  +  bi)  ranges  over  Bk  precisely  once  if  (bi,  •  •  •  ,  bk)  ranges  over  Bk  precisely 
once;  so  that  Nd(cm)  =  We  henceforth  restrict  ourselves  to  SPC  codes  and 
we  will  work  with  Nd,  although  some  of  the  results,  for  example,  lemma  3,  can 
be  applied  to  any  code. 

Lemma  1.  For  any  0,  n,  k ,  d  satisfying  0  >  1,  n  ^  2,  1  ^  k  g  n,  1  g  d  ^  n, 
2/iere  is  a  Bp(n,  k,d)  (Z’S  with  Q{B$(n,  k ,  d)}  ^  1  —  (1/0)  sncft  /or  an?/ 
a  =  (ai,  a2,  •  •  •)  £  B^n,  fc,  d)  n-SPC  code  of  rate  k/n  which  is  generated  from 


,  dn-i  be  random 


ai,---,  an_i  satisfies  Nd  ^  02fc_n^J. 

Proof.  Let  0,  n,  k,  d  be  given  and  let  5i, 
bits,  that  is, 


P(5i  —  bi,  *  •  •  ,  bk  —  bk,  d\  —  ah  *  *  •  ,  dn— i  —  an—\)  —  ^2^ 


for  any  bits  bh  •  •  •  ,  bk ,  ax,  •  •  •  ,  a„_i.  Define  the  random  variables  bk+ 1,  •  •  *  ,5* 
just  as  in  the  definition  of  an  SPC  code,  that  is, 

(17)  =  ar5i  +  dr+iE2  +  •  -  •  +  dr+kSk,  1  ^  r  ^  n  —  fc. 
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Now  let  E  =  (Ei,  •  •  •  ,  5 fc),  5'  =  (E*+i,  •  *  •  ,  En),  and  0  =  (0,  0,  •  •  •  ,  0).  Clearly 
P{S'  =  0|5  =  0}  =  1. 

We  first  prove 

(i)  If  0  *  b  E  B\  V  G  Bn~k  then  P{ S'  =  b'|B  =  &}  =  1/2"-*.  That  is,  if  we 

are  given  E  =  b  9*  0  then  Bu+u  •  •  •  ,  5„  are  random  bits.  Fix  6^0  and  let 

P'{-}  =  P{.|E  =  b}  be  the  distribution  conditioned  on  E  =  6.  Thus  we  wish  to 
prove  that  P'{E'  =  5'}  =  1  /2n— *  for  all  V  G  Let  bi  be  the  last  coordinate 
of  b  which  is  1,  so  that  bi  =  1,  bi+i  =  •  •  •  =  bk  =  0,  and 

(18)  Efc+r  =  Cirbl  +  «r+l?>2  +  *  *  *  +  dr+Z-2&i-l  +  Gr+Z-1* 

Now 

(19)  P'{E*+i  =  bt+i\di  =  ai,  •  *  •  ,  a^_i  =  az_i} 

—  Pf  {d\bi  +  •  •  •  +  dz-iBz-i  -\~  di  =  E*+i|ui  =  fli,  *  *  *  ,  dz-i  =  az-i}  =  2 

for  any  bits  E*+i,  ai,  •  •  •  ,  az_i,  because  the  addition  is  mod  2.  Thus  P'(E*+i  =  0)  = 
P'(Efc+1  =  1)  =  1/2  and  in  order  to  prove  (i)  it  will  be  enough  to  show  that 

(20)  P'  {5fc_j_r  =  Et4-r|E/;+i  =  + 1,  •  •  •  ,  Efc-}-r-l  —  &*+r-l}  =  2 

for  all  bf  G  and  all  r  with  2  ^  r  ^  n  —  k.  But  E&+1,  •  •  •  ,  Efc+r-i  are  functions 
of  only  di,  •  •  •  ,  di+r-2  so  that  it  is  enough  to  show  that 

(21)  P'{E*+r  =  Ei+r|di  =  fli,  •  •  •  ,  dz+r_2  =  az+r-2}  =  2 

for  all  bits  E*+r,  ai,  •  •  •  ,  az+r-2  and  all  r  with  2  g  r  g  n  —  /c.  But  this  is  clearly 

true  from  (8)  so  that  (i)  is  proved. 


Clearly  (i)  implies  that 


(22) 

P{( 5,  5')  =  0}  =  ^ 

(23) 

P{(5, 50  =  (5, 50}  =  ^ 

if  5^0. 

If  Tn 

=  w[(5i,  • 

•  •  ,  E„)]  then 

(24) 

^0 

*5 

a 

II 

HA 

ids' 

because  P{Ei  = 

bh  ■■■  ,bn  =  bn}  =  1/2",  or  0,  for  any 

•  (5i, 

•  •  •  ,  5„)  with 

w[(bh 

•••  ,  bn)] 

=  d.  Let  AI  C  P"-1  consist  of  those  a  such 

that 

(25) 

P{Tn  =  d\a  =  a)  ^  /3P{7\.  =  d}, 

where 

&  =  (di, 

•  •  ,  a„_i),  so  that 

(26) 

P{Tn  = 

d}  ^  P{Tn  =  d|d  G  AJ}P{d  G  AS}  ^  0P{P»  = 

d}P(d  G  AJ} 

hence 

P{o  G  ^1/0.  Thus  there  exists  a  set  A p 

C  Bn 

_1  such  that 

p{ae 

:  =  1 

—  1/0  and  such  that 

/  \ 

(27) 

P{T«  =  d|d  =  a}  g  0P{P„  =  d}  g  fi  n) 

1 

2n 
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for  all  a  £  Ap.  But  this  is  just  the  conclusion  of  lemma  1  with  Bp(ny  k ,  d)  =  the 
set  of  a'  —  (ai,  a2,  •  •  •)  £  25  such  that  (ai,  •  •  •  ,  an_0  £  Ap ;  because,  if  we  take 
such  an  a'  and  generate  the  n-SPC  code  of  rate  k/n  from  it,  then 

(28)  P{Tn  =  d\a  =  o}  = 

so  that  lemma  1  is  proved. 

Lemma  2.  For  am/  0  >  1  £/iere  is  a  set  Bp  C  25  with  Q(Bp)  ^  1  —  (1/0)  such 
that  for  every  a  =  («i,  a2,  •  •  •)  £  #0  the  following  is  true :  For  any  n,  fc  satisfying  1  ^ 
A;  n,  n  ^  2,  J/ie  n-SPC  code  of  rate  k/n  which  is  generated  from  ah  a2,  •  •  •  ,  a„_i 
satisfies 


(29)  g  finA2k~n 

Proof.  Using  lemma  1,  let 

(30) 
so  that 


for  all  d,  1  ^  d  ^  n. 


Bp  =  n  n  n  ^(n,  fc,  d) 

n  =2  *  =  1  d  =  1 


(31)  Q(B*)  ^1-IEE  Q[BUn ,  M)]^-EEEr-4 

n  A:  d  P'1 


n  k  d 


CO  1  1 

=  1-  V  —  >  1  -  - 
^  /Jn2  -  0 


n  =  2  I 


and  lemma  2  is  proved. 

Given  an  n-code  Cn  let  Md  =  the  number  of  h  £  Bn  such  that  w(b)  =  d  and 
such  that  there  exists  ac£Cn  with  be  ^  d.  Recalling  the  decoder  definition  (4) 
and  assuming  d  ^  nph ,  for  the  moment,  we  see  that  Md  equals  the  number  of 
b  £  Bny  of  weight  d,  which  will  not  be  decoded  as  0.  Let  Vn(d)  —  the  volume 
of  a  sphere  of  radius  d  in  Bn ,  that  is, 

@2)  VM  -Eo(:> 

Lemma  3.  Let  Cn  be  an  n-code  such  that  for  all  d,Ni  ^  A  (^).  Then  for  all  d, 
Mi  ^  A  (j)  V„(d). 

Proof.  Let  bi,  be  an  indexing  of  all  the  elements  of  Bn  such  that 
(33)  w(bif)  =  i,  i  =  0,  1,  •  •  •  ,  n;  j  =  1,  2,  •  • 

Let  S(x)  =  1  for  0  ^  x  g  d,  S(x)  =  0  otherwise,  where  d  is  fixed.  Let 

(3) 


■(?> 


(34) 


Cij  —  Y!  S[w(bdic  +  b{j)~\ 


k  —  1 


and  notice  that  e,;  is  independent  of  j.  This  is  intuitively  clear  from  the  geometric 
interpretation  and  may  be  proved  by  observing  that  w{bdk  +  bi,)  is  invariant 
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under  a  simultaneous  permutation  of  the  coordinates  of  both  bdk  and  In  the 
summations  the  range  of  i  is  0,  1,  •  •  n;  the  range  of  j  is  1,  2,  •  •  •,  (l) and 

the  range  of  k  is  1,  2,  •  •  •,  (^J- 

Let  dij  =  1  if  bij  G  Cn  and  0  otherwise,  so  that  Ni  =  E!;  =  A  Now  if 

there  is  a  c  G  Cn  such  that  cbdk  ^  d,  then 
(35)  E  E  S[w(bdk  +  bij)]5ij 

i  j 

will  bo  1  so  that  M,i  S  the  summation  over  k  of  (35).  Interchanging  the  order 
of  summation  we  have  M, j  ^  Z<  Zj  but  e,-j  is  independent  of  j,  so  that 


(36) 


-lb,  ^  Z  elV  Z  ^  -1  Z  (?)  =  -1  Z  Z 

i  j  i  V  /  *  7 

=  A  z  Z  Z  S[w(bdk  +  btj)}. 

k  i  j 


But  ( bdk  +  bi3)  ranges  over  Bn  precisely  once  as  i,  j  cover  their  range  so  that 
(37)  E  E  S[w(bdk  +  b{j)]  =  L  E  S[w(bis)]  =  Vn(d) 

X  j  X  3 

and  lemma  3  is  proved. 

3.  The  proof  of  theorem  4 

Now  theorem  4  follows  quite  directly  from  lemmas  2  and  3  but  before  proceed¬ 
ing  to  the  proof  we  collect  together  some  needed  analytical  details  in 

Lemma  4.  (a)  7/  P{Sn  =  d}  —  (^jpdqn~d  and  up  ^  d  ^  n  then 
\  h>g  p\sn  ^  d)  ^  II  (d)  +  log  p  +  (l  -  d)  log  q. 


Log  VM 


s"(») 


(38) 

(b)  If  0  ^  rf  ^  n/2  then 

(39) 

(c)  If  1  g  d  ^  «/2  then 

(40)  F„(d)  ^  -  l)  Vn(d  -  1). 

Proof,  (a)  If  t  ^  0  then 

(41)  P{Sn  -  d  £  0}  g  -  /*(«) 

where /(0  =  exp  [—  (d/n)t](pel  +  #).  Setting  d//d£  =  0,  the  solution  to  is  given  by 

din  q 


ei0  — 


(42)  -  -  1 

1  —  d/n  p 

and  part  (a)  follows  by  evaluating  f(t0). 


^  1 
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(b)  If  t  ^  0  then 
(43) 

where  g(t)  =  exp  [t(d/n)][  1  +  exp  (—<)]•  Setting  dg/dt  =  0,  the  solution  to  is 
given  by 

(44)  e*  =  -  1  g  1 

a 


and  part  (b)  follows  by  evaluating  g(to). 

(c)  From  Feller  [4],  p.  140,  (3.6)  with  p  =  1/2,  it  follows  that 


(45) 

so  that 

(46) 


VM 
F„(d  ~  1) 


1)8  G-l)»-  LX\ 

_  1  +  _Q_  >  ,  ,  (’<<)  «-M  +  3 

^  F„(d  -  1)  =  ^  ^  n  ^  n  -  d  +  2 


_  i  w  —  d  +  1  n  —  2d  +  3  _  n  —  d  +  1  /  _  d  —  1  \ 

-1+  d  n-d  +  2_1+  d  V  »  -  d  +  2) 


S  1  + 


n  —  d  +  1 


(l _ * - \  =  *±±  _  i 

V  n  —  d  +  l)  d 


and  lemma  4  is  proved. 

Proceeding  to  the  proof  of  theorem  4  we  wish  to  exhibit  a  Bp,  and  naturally  we 
use  the  Bp  whose  existence  is  guaranteed  by  lemma  2.  Let  a  =  (oi,  a2,  •  •  • )  £  Bp, 
n  S  2,  1  g  nR„  =  n[l  —  H (pi)]  =  H  n,  and  let  Cn  be  the  n-SPC  code  of  rate 
fc/n  which  is  generated  from  a1(  •  •  •  ,  a„_i.  Let  p,  p'„  satisfy  (11)  so  that  p  <  p'„  < 
V pi  (V p  +  Vg),  and  let  r  be  the  unique  integer  such  that 


(47) 


V'n  n  <  n  =  < 


V  p 

Vp  +  Vg 


Label  the  codeword  0  by  cx  and  let  Dx  be  given  by  (4)  so  that 


(48)  P{Di|0}  ^  L  McpV'"  +  P{Sn  ^  r  +  1} 

d  —  1 

and  from  lemmas  2  and  3 

(49)  P{Z>(|0}  g  /3n*2-»W*  (j)  pV_<iF„(d)  +  P{Sn  ^  r  +  1}. 

Now,  iVd(cm)  is  independent  of  m,  and  lemma  3  can  be  apphed  to  cm,  as  well  as 
to  0,  so  that  the  same  bound  holds  for  P{Dm\cm}  and  hence  for  ep(Cn).  If 
1  g  d  ^  r  then  from  lemma  4(c)  and  (47). 


METRIC  STRUCTURE  OF  CODES 


679 


(fW ~dVn(d) 

(50)  7  »  \  - 

(d  “  1)p“-y-<wyn(d  -  l) 

so  that  the  terms  in  the  summation  are  increasing  and  since  there  are  at  most  n/2 
of  them  we  get 

(51)  ep(Cn)  g  |  n6  2-"«»  (”)  pY-rF»(r)  +  P{S„§r+l). 

But  2~H(v')n  Vn{r)  g  1  by  lemma  4(b)  and  (47)  so  that  using  P{Sn  ^  r}  ^ 
P{Sn  §  npi  -  1}  we  get 

(52)  ep(Cn)  ^  |  n6P{Sn  S  np'n  -  1} . 

Now  lemma  4(a)  and  H\jpfn  —  (1/n)]  ^  if(p£)  imply  that 

(53)  i  log  ep(C„) 

=  ;llog2 n6  +  H  (*•  ~ »)  +  (p»  -  £) logp  +  (i  -  +  £)  lQg« 

-  n log  2  p  +  +  p" log  P  +  t1  “  Vn)  log  q 

and  theorem  4  is  proved. 
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